‘other AI researchers are going to kill everyone, and we are the only hope of humanity’
Let me explicitly reject such rhetoric then.
The difficulty of safety is uncertain: it could be very easy for anyone with little time, or it could be quite difficult and demand a lot of extra work (which might be hard to put in given competitive pressures). The region where safety depends sensitively on the precautions and setup of early AI development (from realistic options) should not be much larger than the “easy for everyone region,” so trivially the probability for building AI with good outcomes should be distributed widely among the many possible AI building institutions: software firms, government, academia, etc. And since a small team is very unlikely to build AGI first, it can have at most only a very small share of the total expected probability of a good outcome.
A closed project aiming to build safe AI could have an advantage either by using more demanding safety thresholds and by the possibility of not publishing results that require additional work to make safe but could be immediately used for harm. This is the reasoning for classifying some kinds of work with dangerous viruses or nuclear technology or the like. This could provide some safety boost for such a project in principle, but probably not an overwhelming one.
Secrecy might also be obtained through ordinary corporate and government security, and governments in particular would plausibly be much better at it (the Manhattan Project leaked, but ENIGMA did not). And different safety thresholds matter most with respect to small risks (most institutions would be worried about large risks, whereas those more concerned with future generations might place extra weight on small risks). But small risks contribute less to expected value.
And I would very strongly reject the idea that “generic project X poses near-certain doom if it succeeds while project Y is almost certain to have good effects if it succeeds”: there’s just no way one could have such confident knowledge.
And the progress is so disappointing perhaps thanks to P!=NP and the like—the tasks do not have easy general solutions, or even general purpose heuristics.
You can still get huge differences in performance from software. Chess search explodes as you go deeper, but software improvements have delivered gains comparable to hardware gains: the early AI people were right that if they had been much smarter they could have designed a chess program to beat the human world champion using the hardware they had.
Part of this is that in chess one is interested in being better than one’s opponent: sure you can’t search perfectly 50 moves ahead, but you don’t have to play against an infinite-computing-power brute-force search, you have to play against humans and other computer programs. Finance, computer security, many aspects of military affairs, and other adversarial domains are pretty important. If you could predict the weather a few days further than others, you could make a fortune trading commodities and derivatives.
Another element is that humans are far from optimized to use their computation for chess-playing, which is likely true for many of the other activities of modern civilization.
Also, there’s empirical evidence from history and firm R&D investments that human research suffers from serial speed limits of human minds, i.e. one gets more progress from doubling time to work than the size of the workforce. This is most true in areas like mathematics, cryptography, and computer science, less true in areas demanding physical infrastructure built using the outputs of many fields and physically rate-limited processes. But if one can rush forward on those elements, there would then be an unprecedented surge of ability to advance the more reluctant physical technologies.
Let me explicitly reject such rhetoric then.
The difficulty of safety is uncertain: it could be very easy for anyone with little time, or it could be quite difficult and demand a lot of extra work (which might be hard to put in given competitive pressures). The region where safety depends sensitively on the precautions and setup of early AI development (from realistic options) should not be much larger than the “easy for everyone region,” so trivially the probability for building AI with good outcomes should be distributed widely among the many possible AI building institutions: software firms, government, academia, etc. And since a small team is very unlikely to build AGI first, it can have at most only a very small share of the total expected probability of a good outcome.
A closed project aiming to build safe AI could have an advantage either by using more demanding safety thresholds and by the possibility of not publishing results that require additional work to make safe but could be immediately used for harm. This is the reasoning for classifying some kinds of work with dangerous viruses or nuclear technology or the like. This could provide some safety boost for such a project in principle, but probably not an overwhelming one.
Secrecy might also be obtained through ordinary corporate and government security, and governments in particular would plausibly be much better at it (the Manhattan Project leaked, but ENIGMA did not). And different safety thresholds matter most with respect to small risks (most institutions would be worried about large risks, whereas those more concerned with future generations might place extra weight on small risks). But small risks contribute less to expected value.
And I would very strongly reject the idea that “generic project X poses near-certain doom if it succeeds while project Y is almost certain to have good effects if it succeeds”: there’s just no way one could have such confident knowledge.
You can still get huge differences in performance from software. Chess search explodes as you go deeper, but software improvements have delivered gains comparable to hardware gains: the early AI people were right that if they had been much smarter they could have designed a chess program to beat the human world champion using the hardware they had.
Part of this is that in chess one is interested in being better than one’s opponent: sure you can’t search perfectly 50 moves ahead, but you don’t have to play against an infinite-computing-power brute-force search, you have to play against humans and other computer programs. Finance, computer security, many aspects of military affairs, and other adversarial domains are pretty important. If you could predict the weather a few days further than others, you could make a fortune trading commodities and derivatives.
Another element is that humans are far from optimized to use their computation for chess-playing, which is likely true for many of the other activities of modern civilization.
Also, there’s empirical evidence from history and firm R&D investments that human research suffers from serial speed limits of human minds, i.e. one gets more progress from doubling time to work than the size of the workforce. This is most true in areas like mathematics, cryptography, and computer science, less true in areas demanding physical infrastructure built using the outputs of many fields and physically rate-limited processes. But if one can rush forward on those elements, there would then be an unprecedented surge of ability to advance the more reluctant physical technologies.